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Abstract:  Many  high-end  deployed  military  and  commercial  applications  share  a  common  need 
to  achieve  high  to  very  high  compute  and  bandwidth  density  in  the  smallest  possible  volume.  In 
addition,  deployed  military  applications  layer  on  additional  environmental  requirements  such  as 
higher  levels  of  shock,  vibration,  endurance  vibration,  temperature,  and  condensing  humidity. 
Each  of  those  adds  constraints  on  the  solution  space  for  maximizing  compute  and 
communication  density. 

Not  all  HPEC  applications  can  use  the  same  solution  due  to  varying  limits  on  total  size  or  weight 
and  varying  levels  of  ruggedness.  For  example,  the  size  and  weight  requirements  differ  greatly 
for  manned  surveillance  aircraft,  large  UAYs,  and  small  UAVs.  This  presentation  will  explore 
different  options  for  high-density  system  designs  while  meeting  the  requirements  for  each  of 
these  applications. 
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Figure  1.  Power  flux  of  recent  and  near-term  PowerPC  processors. 
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A  big  obstacle  is  the  well-known  trend  of  increasing  power  consumption  for  newer  faster 
processors.  Faster  processors  require  faster  memory,  faster  interconnect,  and  more  I/O.  Not  as 
well  known  is  the  power  increase  due  to  the  move  to  high-speed  switch  fabrics  clocked  up  to 
multiple  GHz  and  memory  systems  being  clocked  well  into  the  hundreds  of  MHz.  A  high-speed 
switch  fabric  and  memory  system  exacerbate  the  thermal  problem  even  as  it  solves  bandwidth 
problems.  In  these  cases,  it  is  the  large  number  of  components  and  the  concentrated  point  sources 
of  power  that  create  the  biggest  challenge  beyond  that  of  the  overall  increasing  power  and 
shrinking  die  size.  This  combines  with  the  increasing  power  consumption  to  create  a  significant 
challenge.  Figure  1  shows  a  comparison  of  past  and  present  processor  power  flux. 

Yet  these  are  not  the  only  demanding  components  in  the  system.  FPGAs,  SRAM,  on-board 
power  supply  FETs,  integrated  DC/DC  converters,  and  various  ASICs  must  all  be  examined  as 
point  sources  of  heat  in  today’s  designs.  Previous  designs  have  70%  or  more  of  the  power  in  the 
system  going  to  the  processors  alone.  For  current  designs,  it  is  typically  40-50%,  with  the 
balance  going  to  areas  such  as  interconnect,  memory,  and  supporting  components.  Large  FPGAs 
used  for  computing  can  consume  20W,  and  the  power  supply  FETs  can  have  the  power  flux 
(W/cm2)  of  a  similar  magnitude  to  the  processors. 

When  component  package  type  is  taken  into  account  (e.g.,  plastic  TSOP  for  DDR-DRAM, 
PBGA  for  ASICs),  the  other  components  mentioned  can  approach  the  same  power  and  thermal 
management  challenge  as  the  processor.  Designing  a  system  to  optimize  the  performance  density 
with  respect  to  each  of  these  is  challenging. 

For  any  given  selection  of  processor,  fabric,  and  support  chips,  the  areas  of  system  design  that 
can  be  worked  to  maximize  overall  power  density,  and  therefore  performance  density,  are  largely 
driven  by  the  mechanical  aspects  of  the  board  form  factor  and  the  resulting  cooling  methods  for 
the  boards  and  chassis. 

Traditionally,  the  choices  for  cooling  methods  have  been  convection-cooled  (air-cooled)  and 
conduction-cooled.  Recent  years  have  seen  a  growing  interest  in  spray-cooling  (evaporation- 
cooled),  because  it  offers  a  greater  potential  for  thermal  dissipation  than  either  air-cooled  or 
conduction-cooled  while  providing  the  benefits  of  a  sealed  enclosure  for  environments  with  little 
air,  dirt,  or  corrosive  elements.  Yet  when  the  size,  weight,  and  maturity  of  the  various  cooling 
methods  are  examined,  air-cooling  is  shown  to  meet  the  right  set  of  tradeoffs  against  the 
commercial  and  military  requirements  for  much  of  the  HPEC  space. 

This  presentation  will  describe  one  technique  for  extracting  greater  thermal  efficiency  from  an 
air-cooled  design  we  call  “finely  managed  air.”  In  this  approach  -  covers  on  the  boards,  heat¬ 
sinks  captive  in  these  covers,  and  airflow  shaping  in  the  covers,  slots,  and  chassis  inlet  areas  - 
are  all  designed  together  as  a  system  to  carefully  direct  all  available  air  flow  over  the  hottest 
components  to  extract  the  maximum  thermal  efficiency.  While  in  conventional  board  and  heat 
sink  design  the  majority  of  the  air  wants  to  flow  around  the  high-impedance  heat  sinks  instead  of 
through  them,  the  new  approach  uses  features  in  the  cover  that  directs  air  through  the  heat  sinks. 
Since  high  velocity  flow  tends  to  ride  up  the  backplane  and  starve  the  front  of  the  board,  there  is 
also  a  need  to  balance  the  airflow  front  to  back  within  a  slot  to  also  maximize  cooling  efficiency. 
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Features  built  into  the  cover  achieve  this  in  a  way  to  have  the  minimum  effect  on  overall 
pressure  drip. 

Such  “finely-managed  air”  designs  have  been  verified  using  both  Computational  Fluid  Dynamics 
(CFD)  modeling  and  lab  measurements  of  operational  hardware  at  the  component,  board,  and 
system  level,  and  this  paper  will  present  the  analysis  of  that  data.  In  addition  to  the  airflow¬ 
shaping  aspects  of  the  cover,  the  cover  also  adds  a  very  important  ruggedizing  structural  element 
to  the  board  while  not  penalizing  the  available  surface  area  of  the  PWB  with  stiffeners,  mounting 
holes,  and  keep-out  regions. 

To  further  maximize  the  effect  of  such  an  approach,  each  processor  or  high-power  component 
should  get  an  independent  flow  of  air.  Currently  high-end  deployed  commercial  off-the-shelf 
(COTS)  high-density  designs  use  large  area  boards,  such  as  VME  9U  x  400mm,  to  maximize  the 
number  of  processors  per  given  volume  using  both  the  available  height  and  depth.  New 
commercial  form  factors  have  similar  approaches,  such  as  PICMG3.0  with  an  8U  high  board  and 
less  deep  at  280mm.  In  those  tail-board  designs,  components  not  on  the  inlet  edge  of  the  board 
will  see  air  that  was  already  heated  by  processors  or  other  high-power  components  lower  on  the 
board.  There  can  be  as  many  as  three  to  six  rows  of  high-power  components  in  an  attempt  to 
maximize  processing  physically  in  the  layout. 

In  the  face  of  these  thermal  considerations,  concurrent 
engineering  must  be  given  to  board  and  chassis  design. 

The  optimal  location  for  high-power  components  is 
along  the  bottom  of  the  board  on  the  leading  edge  of  the 
airflow.  A  long,  shorter  board  therefore  provides  the 
maximum  thermal  dissipation  for  a  system  by 
combining  this  longer  leading  edge  with  a  lower  flow 
resistance  due  to  the  shorter  nature  of  the  board.  In 
order  to  get  the  highest  density  in  a  given  vertical 
space,  two  boards  stacked  vertically  with  an  air  intake 
in  the  middle  provide  a  significant  improvement  over 
past  practice  in  the  thermal  dissipation  capacity  for  a 
given  board  set.  An  example  of  such  a  configuration  is 
shown  in  Figure  2.  For  both  boards,  the  hottest 
components  are  placed  on  the  leading  edge  of  the 
airflow. 

This  organization  allows  for  continued  use  of  a  large 
common  backplane.  A  large  common  backplane 
remains  the  linchpin  to  achieving  tens  of  GB/s  of  inter¬ 
chassis  bandwidth  in  a  very  compact  and  very  high-speed  signaling  environment. 

This  presentation  will  discuss  such  design  trade-offs  for  different  deployed  HPEC  environments 
in  terms  of  size,  power,  and  weight  constraints. 
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PloiAl 


Figure  2.  Two  short 
boards  mounted  vertically 
sharing  the  same  air  intake 
to  minimize  vertical  space. 
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Many  high-end  deployed  military  and  commercial  HPEC  applications 
share  a  common  need  to  achieve  high  to  very  high  compute  and 
bandwidth  density  in  the  smallest  possible  volume. 

In  addition,  deployed  military  applications  layer  on  additional 
environmental  requirements  such  as  higher  levels  of  shock,  vibration, 
endurance  vibration,  temperature,  and  condensing  humidity.  Each  of 
those  adds  constraints  on  the  solution  space  for  maximizing  compute 
and  communication  density. 

Not  all  HPEC  applications  can  use  the  same  solution  due  to  varying 
limits  on  total  size  or  weight  and  varying  levels  of  ruggedness.  For 
example,  the  size  and  weight  requirements  differ  greatly  for  manned 
surveillance  aircraft,  large  UAVs,  and  small  UAVs. 


This  poster  presentation  explores  two  of  the  many  key  dimensions  to 
achieving  high  functional-density  systems: 

I  Thermal  management 
I  Board  real-estate  utilization 
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rThe  power  flux  (W/cm2)  of  the  PowerPC  processor  continues 

to  increase. 


l  l  l  l  l 

PPC7410  PPC7450  P PC  7455  P PC  7457  P PC  7457  PPC970 
500  MHz  700  MHz  1GHz  1GHz  1.3  GHz  1.8  GHz 


Data  taken  from  publicly  available  sources  such  as 
data  sheets,  physical  measurements,  and  articles 


Similar  increases  are  being  seen  in  FPGA  compute 

elements. 
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Non-processor  power  is  increasing,  too 

I  In  addition  to  the  increasing  power  flux  of  processors,  non¬ 
processor  components  in  these  COTS  designs  must  be  thermally 
managed  like  never  before. 

I  While  the  PowerPC  has  been  -80%  of  a  processing  nodes  total 
power,  it  is  now  closer  to  50-60%  with  the  balance  of  the  power 
dissipation  coming  from  other  components  of  the  node;  memory, 
control  and  interconnect  ASICs,  FPGAs,  and  DC/DC  power 
converters. 


•  Modeling  the  future 


I  The  next  few  slides  contrast  the  past  with  the  present  and  future 
in  the  areas  of  thermal  management.  These  CFD  modeling  results 
are  from  a  customer-driven  study  with  results  correlated  against 
actual  similar  hardware  under  operational  test. 

I  For  visual  simplicity,  these  are  modeled  as  straight-thru  air-flow. 
In  a  rack-mount  system  with  front-to-back  flow  where  air  is  taking 
a  90°  turn  in  and  a  90°  turn  out,  resulting  temperature  may  be 
10-20+C  higher  without  proper  chassis  level  management. 
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Very  high  routing  and  component  density 
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Past  thermal  management  solutions  were  fine  for  commercial 
designs  and  had  headroom  for  extending  designs  to  MIL  deployed 


Models  for  8W  PowerPC 


Commercial:  35C  inlet;  5k  feet 
with  “coarsely”  managed  air 


Military:  55C  inlet;  10k  feet 
with  “coarsely”  managed  air 
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The  past: 

•  “Life  was  good”  for 
commercial 
environments... 

•  And,  headroom  was 
available  for  moving  to 
MIL-deployed 
environments  such  as 
55C  concurrent  with  10K 
altitude 


Ideal  flow  shown  in  this  study  -  temperatures  may  increase 
an  additional  10-20+C  in  a  non-optimized  chassis. 
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New  processors  push  even  the  commercial  designs  to 
meet  spec  and  make  MIL-deployed  derivatives 
unattainable  using  past  techniques 


Models  for  20W  PowerPC 
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Commercial:  35C  inlet;  5k  feet 
with  “coarsely”  managed  air 


Military:  55C  inlet;  10k  feet 
with  “coarsely”  managed  air 
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Today's  problem: 

Even  with  each 
processor  getting 
directed  air  at  the  inlet 
temperature  (l.e.,  no 
preheating),  the  new 
generation  of  processors 
have  little  to  no  margin 
against  their  Tj  or  Tc  max 
temperatures  in  MIL 
deployed  environments 

New  solutions  are 
required! 


Ideal  flow  shown  in  this  study  -  temperatures  may  increase 
an  additional  10-20+C  in  a  non-optimized  chassis. 


“Finely  Managed”  air  at  the  board  level  explicitly  shapes  and 
tunes  airflow  with  precise  control  over  impedance  drop  and 
impedance  distribution  over  the  boards  surface 


Models  for  20W  PowerPC 


Commercial:  35C  inlet;  5k  feet 
with  "finely”  managed  air 


Military:  55C  inlet;  10k  feet 
with  “finely”  managed  air 
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New  solution: 

•  To  allow  use  of  the  new 
generation  of  processor, 
memory,  ASIC  components 
-  new  airflow  shaping  and 
management  techniques 
are  required  to  achieve 
similar  temperatures  to  the 
past  commercial  designs 

•  These  same  techniques 
enable  use  of  these  next- 
generation  components  in 
MIL-deployed  environments 
with  only  a  little  headroom 


Ideal  flow  shown  in  this  study  -  temperatures  may  increase 
an  additional  10-20+C  in  a  non-optimized  chassis. 


Future: 

•  More  aggressive  use  of  these 
techniques  throughout  the  chassis  can 
result  in  even  more  benefit 

•  If  these  improvements  weren’t  achieved, 
processing  and  memory  frequencies 
would  be  reduced  by  1/2  to  1/3  -  l.e.,  400 
to  600  GFLOPS  (peak)  at  the  chassis  level 


Model  inputs: 

•  Processor:  20W  PowerPC 

•  Other  devices  w/  representative 
power 

•  MIL  Deployed:  55C  inlet.  10K  alt. 

>  Finely  managed  airflow,  more 
aggressive  throughout  the  chassis 
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r  Aside  from  thermal  management,  use  of  board  real  estate  has  also 
become  a  huge  challenge  to  functional-density. 

•  The  mechanical  features  such  as  board  stiffeners,  rugged  heat-sink 
mounting,  and  such  for  MIL-rugged,  air-cooled  boards  takes 
precious  inches2  of  board  space.  MIL  requirements  for  board-  and 
system-level  endurance  vibration  and  shock  pulses  drive  many  of 
the  rugged  structural  requirements. 

•  The  holes,  pads,  and  keep-outs  for  these  items  was  now  causing 
significant  loss  of  density  -  for  both  internal  routing  and 
components. 

•  Example:  Due  to  routing  and  placement  restrictions,  using 
conventional  techniques  yielded  one  less  processor  node  on  the 
board  format  under  study,  or  24  less  nodes  at  the  chassis  level. 
Thus  approximately  250  GFLOPS  (peak}  would.  be  left  on  the  table 

using,  conventional  technigues. 

•  Innovation  for  forced  air-cooled  boards  was  required  and  resulted  in 
putting  thermal  management  and  structural  rigidity  features  in  the 
Z-dimension  via  a  cover  mechanism  ...  10 
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Heat  S ink 


The  cover  provides: 

I  Rugged  mounting  points  for  heat 
sinks 

I  Rugged  structural  members  that 
the  board  mounts  into 

I  Facilities  for  finely  managed  air 
features 

•  In  the  cover  surface  itself 

•  On  the  inlet 

•  On  the  outlet 

•  At  the  interface  to  the  card-cage 

I  Finely  controlled  air  plenums  are 
created  at  a  slot-by-slot  level  at 
the  board  and  card-cage 
structures 


These  features  can  be  tuned  as 
platform  and  application 
requirements  drive  needs 


Module  inlet  view 


Patent  pending 
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Evaluation  done  using 
NAVMAT  P-9492“Willougftby” 

A  typical  random  vibration 
test  for  components  and  PWBs 


o.ooi 


Full  Cover 
f = 138hz 
d  =  0.032 


Patent  pending 


Performance  of  full  cover 
using  the  Z-dimension  is 
similar  to  or  better  than 
other  structural  methods 
that  take  up  significant 
board  space  and  are 
often  at  odds  with  the 
direction  and  amount  of 
airflow  required. 


80  350 
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2000 
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Bare  pwb 
f  =  52hz 
d  =  0.084 


Ribs  parallel 
to  air  flow 
f  =  58hz 
d  =  0.076 
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Patent  pending 


Full  Cover 
f = 138hz 
d  =  0.032 


Ribs  backside 
blocking  air  flow 
f  =  72hz 
d  =  0.066 


Typical  goal  for  this 
style  of  module  is: 

•  125Hz  <f<  150Hz 

•  0.025  <  d  <  0.040 
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r*  For  a  total  system  solution,  solving  these  present 
day  board-level  thermal  and  structural  management 
isn’t  the  whole  issue,  chassis-level  functional 
density  must  complement  these  solutions. 

•  The  balanced  HPEC  TFLOP  under  study,  in  a  small 
volume  chassis,  would  need  all  of  this  supporting 
cast  in  addition  to  the  processors: 

I  >100  GB  total  memory 

I  >50GB/s  aggregate  and  bisection  backplane  interconnect 
I  10-20GB/S  aggregate  and  bisection  inter-chassis  interconnect 

I  10-20GB/S  concurrent  external  I/O 

(e.g.,  streaming  sensor  data  over  fiber,  VITA-1 7.1 ) 

I  2-3  dozen  open  standard  I/O  slots  or  sites 

(e.g.,  IEEE  1386.1  PMC,  VITA-42  XMC) 
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This  picture  represents  the  results 
of  one  study  which  determined  a 
means  to  meet  these  high 
functional-density  requirements 
(e.g.,  “a  balanced  TFLOP”)  in  a 
MIL-deployable  HPEC  system. 


Two  processor  modules  mounted  vertically  sharing 
the  same  air  intake  to  minimize  vertical  space. 


RapidIO  backplane  interconnect  w/  >50GB/s  Patent  pending 

aggregate  and  bisection  bandwidth 


This  arrangement  achieves  two 
leading  edges  of  inlet  air  to  enable 
high-density  without  having  to 
have  “columns”  of  the  hot  Air 

components,  such  as  processors, 
heating  each  other. 


From  the  previous  thermal  study 
presented,  it  is  obvious  that  the 
heating  effects  of  these  new 
generation  components  won’t 
allow  column  organizations  used  in 
the  past. 

Processor  module  under  study 


Approx  21”  H  x  23”  D, 
19”  rack  mountable 
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Mercury  has  invested  in  innovations  towarc 
solving  this  new  class  of  problems  in  the  COTS 
MIL-deployed,  high-density  HPEC  application 
space. 


Back  to  Agenda 


Next  Abstract 


To  date,  8  patents  related  to  the  IP  for 
these  methods  described  have  been  filed.  Five 
of  these  filings  have  "notice  of  allowance" 
which  is  the  last  step  to  patent  registration.  The 
remaining  3  filings  are  in  the  last  stages  of  the 
PTO  process. 


The  IP  represented  by  these  patents  will  appear 
in  Mercury’s  future  COTS  MIL-deployable  forced 
air-cooled  HPEC  products  under  development. 
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